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Abstract 

The Escola Tecnica Superior d'Enginyeria Quimica has a long tradition in the deployment of social competencies 
in engineering curricula through Integrated Projects (IP) carried out in structured teams. Social competencies 
are taught and practiced during the development of the IPs. We conceptually introduce a methodology for a 
360° assessment of the students' social competencies, as a tool to foster the improvement of their competency 
levels. In this article we analyze the results of the pilot test where the aforementioned methodology has been 
implemented in the Bachelor studies of Chemical Engineering. The results indicate that it is possible to 
objectively obtain the student's competency level discriminating among different social competencies, as well 
as among different students in the same team. The application of this tool fosters the development of specific 
educative actions to help the students with low competency profile, to reach acceptable levels for a successful 
insertion in the labor market. 
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1 INTRODUCTION 

The new challenges of the global society demand from higher education graduates capable of leading and 
fostering the progress of the world. The new paradigm aiming at meeting these challenges focuses on 
developing competencies rather than only knowledge and practical skills in the students during their way 
through higher education (ABET, 2012; Spencer & Spencer, 1993; McClelland, 1998; AQU, 2012). 

The idea of competency first appeared from the observation of what makes the difference between average 
and outstanding professionals in different jobs. The birth of the concept can be attributed to David C. 
McClelland (McClelland, 1973), who published the article Testing for Competence Rather Than Intelligence, in 
1973. In that article the author indicates how traditional tests, based on academic aptitude and knowledge, are 
unable to predict job performance or success in life. It is interesting to read the detailed example of the 
recruitment of the U.S. State Department Foreign Service Information Officers (Spencer & Spencer, 1993), as a 
case where these ideas are clearly explained. 

Spencer and Spencer (1993) define competency as (sic): 

A competency is an underlying characteristic of an individual that is causally related to criterion- 
referenced effective and/or superior performance in a job or situation. 

Notice that we use the word competency in the sense of "an important skill that is needed to do a job". It 
should be differentiated from competence, which is "the ability to do something well". 
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Although Spencer's definition is neither unique nor probably the best, it indicates that the individuals that excel 
in their function are not distinguished from the average only by their knowledge or their experience. Other 
capacities, many of them of social nature, also play a very important role. 

Hence, in order to respond to the new paradigm, one has to incorporate social competencies in the engineering 
curricula. A consequence of this fact is the need of providing an adequate feedback to the students. Therefore, 
it is essential to objectively evaluate their competency level during the studies, to push them towards becoming 
outstanding professionals in their future jobs. 

The educational model of the Escola Tecnica Superior d'Enginyeria Quimica (ETSEQ) is based on Integrated 
Projects (IP) in which the students work in structured teams on open design projects related to Chemical 
Engineering and Food Engineering (Witt, Alabart, Giralt, Herrero, Verms & Medir, 2006). In this context, 
students receive specific training about social competencies and interact in an adequate environment to 
practice them. Social competencies are reflected to the individual behavior (Sune & Bonet Avalos, 2012). The 
most significant behaviors will take place in quasi professional situations that occur during the IP. Hence, it is 
essential to observe the behavior of the individuals while they are acting in their roles, in order to determine 
the competency level. Therefore, the intrinsic difficulty of assessing social competencies lies in the need of an 
objective and reliable procedure based on direct observations of the behaviors. 

In this article we discuss the procedure implemented in the ETSEQ to observe and assess these social 
competencies among the students. We also discuss the results of the pilot test carried out in the ETSEQ during 
the second semester of the academic year 2012-2013 of the Bachelor studies of Chemical Engineering (GEQ) 
and Food Engineering (GEA), which show that an objective assessment of social competencies is possible. 


2 METHODOLOGY 
2.1 Background 

From the beginning of 2012, the ETSEQ has developed and implemented an evaluation methodology to assess 
the five social competencies included in its competency dictionary. These five social competencies are 
described in the memoranda of the Bachelor studies (GEQ, 2009; GEA, 2009), and are listed in Table 1. 


Classification 

Competency 

Technical Competencies 

Al. Technical 

A2. Professional 

Social Competencies 

Bl. Human Interaction & Versatility 

B2. Facilitative Leadership 

B3. Teamwork 

B4. Active Learning & Responsibility 

B5. Initiative & Innovation 

Nuclear Competencies 

Cl. Communication 

C2. Social 


Table 1. Competency dictionary of the ETSEQ 


However, these competencies, as they stand, are meaningless if no additional explanation is provided. In Table 
2 we give the definition of competency B1 in terms of behavioral descriptors, which are further deployed 
through a list of observable elementary behaviors related to the latter. We will return to this point in section 
2.3. In summary, the set of social competencies B1-B5 need 22 behavioral descriptors, with 37 elementary 
behaviors, which are ultimately evaluated one by one. 

The methodology proposed here is based on two postulates (Sune & Bonet Avalos, 2012) that set its conceptual 
epistemological basis: 
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Postulate 1. The object accessible to the observer is the so-called observable. Hence, for our purposes, the 
observable is: 

The behavior of an individual in a given moment , in a concrete environment , and exerting a given role 
related to her/his current studies (and the future profession). 

One can not have access to what a person is or to what in psychology is known as a mental state. Instead, one 
can observe what a person is doing as the only indicator of his/her capabilities. 


Postulate 2. The measure is a comparison process. The objectivity of the measure is hence based on the 
existence of a standard, in the first place, which will be the same for all the observers. Secondly, considering 
that human behavior depends on the circumstances and the states of mind (which exist as facts of the inner 
individual experience but are not directly accessible to others), the final objective is, hence, to determine what 
the person does in general, not in a specific moment, a particular day, or under concrete circumstances. 
Therefore, to guarantee the objectivity of the assessment of the competency level, it has to be determined 
from an average of observations obtained from different observers, at different moments, under different 
circumstances, but always while the individual is exerting her/his role, which the standard refers to. 

According to this postulate, it is very important that the students perform quasi-professional roles during their 
education, since this is the only context in which the behaviors related to the social competencies can be 
observed. In the next section we introduce the context in which the assessment methodology has been 
implemented. 


2.2 Context 

The competency assessment has been implemented in the subjects where the IP's, mentioned in the 
Introduction, are carried out. The IP implies building balanced teams using the principal profiles of the students. 
Such profiles can be obtained from questionnaires of the type of Belbin (Belbin, 2014), together with relevant 
information about them like grades, geographic constraints, etc. Within these teams, we differentiate two roles: 
leader and team member. Professors of the first year play different roles. On the one hand, a professor can be a 
consultant and address the questions posed by the students on a given subject. On the other hand, the same 
professor acts as a client because he/she will grade the final report at the end of the IP. Another professor plays 
the role of coach to support the team during the development of the IP, without intervention in the evaluation. 
The goal of the IP is proposed by the coordinators, which act as sponsors. 

The IP of the first academic year involve students of the first year and students of the fourth year, enrolled in 
the elective subject Team Leadership Practice (PLE). In this particular IP, the leader of the team is one of these 
PLE students. He/she should lead from 5 to 7 students of the first year who act as team members (Witt et al., 
2006). The professors of PLE coach the leaders during the IP, grade their performance in the subject, and also 
assess their social competencies. A scheme of the organization of the IP of the first year is shown in Figure 1. In 
the IP of the second and third year, the leader is a student belonging to the same academic year, who also plays 
a significant role as a team member. 

The team has to set a Team Charter in order to determine the objectives, the team rules, the planning of the 
tasks, the deadlines, the deliverables, etc. The leader and team members have to meet at least once a week 
according to a schedule established at the beginning of the academic year. 
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Figure 1. Structure of the teams involved in the IP of the first academic year 
2.3 Implementation of the methodology of evaluation of competencies 

According to Postulate 2, one has to define a standard for each role. The standard has to be related to the 
competency dictionary given in Table 1. Such a standard is deployed as a collection of rubrics, bound to the 
behavioral descriptors, which define each competency, as in Table 2. Hence, the standard materializes in a 
questionnaire to be used for all the observers to assess all the individuals acting within the same role. In our 
case, we have chosen these rubrics to be descriptions of elementary behaviors, also known as pinpointed 
behaviors, easily identifiable in the student's behavior during the IP. 


Competency 

Behavioral descriptors 

Elementary behaviors 

Bl. 

Human 

Interaction 

& 

Versatility 

1.1 Communicates effectively in 
interpersonal and group 
situations. 

Looks for the best in team members, irrespective of 
personal opinion and judgment. 

Expresses her/himself clearly, taking care of being 
understood (for example, expressing ideas, giving 
feedback, addressing tasks to the team). 

Builds effective relationships in and outside the team to 
improve performance. 

Updates team members and stakeholders on progress. 

1.2 Able to translate thoughts into 
oral and written communication. 

Clearly explains to the team members what her/his 
expectations are. 

1.3 Adapts behavior and work 
method in response to changing 
conditions. 

Increases her/his leadership in front of difficulties and 
challenges. 

Encourages others to contribute and come forward with 
own ideas. 

In the moments of high workload, with a positive attitude, 
encourages the team members by acting as a role model. 

1.4 Deals effectively with high 
workloads. 

When the workload is high she/he reviews the work plan 
with the team and resets priorities. 

1.5 Displays resilience. 

Is able to cope with setbacks and disappointments while 
focusing on moving the team forward. 

1.6 Resolves conflict 
constructively. 

In the case of a conflict, she/he faces the situation by 
finding a solution with win-win scenarios. 

Understand conflict resolution methods and applies them 
effectively. 


Table 2. Rubrics of the B1 competency for the leaders. The table shows the competency, the behavioral 

descriptors and the elementary behaviors 
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The students are provided with forms containing the questionnaires with the complete list of competencies, 
behavioral descriptors, and the associated pinpointed behaviors for each role. In these questionnaires the 
pinpointed behaviors have to be evaluated one by one by the observer, using a scale with three values: 0, 1, 2, 
depending on whether a given elementary behavior has been seldom observed (0), sometimes (1), or often (2), 
if it denotes frequency, or low (0), medium (1) or high (2) if it denotes intensity. If circumstances are not 
appropriate for the observation of a specific elementary behavior, the observer leaves the evaluation of the 
item blank. At the end of the year, the average of the observations for each elementary behavior is determined. 
Collecting these average values of the elementary behaviors for a given competency one obtains the 
competency level from their mean on a scale from 0 to 2. The level is finally scaled up to a range between 0 and 
10 for the ease of interpretation. The student finally receives a qualitative assessment of her/his competency 
profile in terms of superior performance (8.25-10 ), average performance (6-8.25 ), needs improvement (4-6), 
and needs serious improvement (0-4). 

The process of evaluation starts with a training session with the students in order to explain to them the 
methodology step by step. From this moment on, the students have access to the rubrics and a detailed guide 
with all the information required. 

The dynamics of the evaluation then proceeds as follows. The team members have to evaluate once a week 
his/her teammates (cross-evaluation), perform the self-evaluation, and also evaluate the leader. To reduce the 
evaluation burden, however, the competencies to be evaluated are distributed among the team members so 
that each team member evaluates all the other team members with respect to one single competency (maybe 
two, depending on the size of the team), while another team member evaluates the leader. These roles rotate 
among the team members according to a calendar that the leader establishes at the beginning of the project. 
The leader has to evaluate individually her/his team members once a week, and do the self-evaluation with the 
same periodicity. The leader is furthermore evaluated twice a month by the PLE professor. In the example of the 
team member, the relative weight of each type of evaluation is approximately the same (1/3). 

3 RESULTS 

To check the feasibility of the procedure, we conducted a pilot test during the 2nd semester of the academic 
year 2012-2013. The pilot test involved one team of the integrated project (GEQ/GEA) of the first year (the 
leader was a student of the fourth year), one team of the fourth year (involving several students of PLE of the 
GEQ) and one team of the second year (GEA). A total of 17 students, 3 of them as leaders, and the rest as team 
members, were involved. The procedure allowed us to collect 70 observations related to the role of leaders and 
232 observations for the role of team members; that is, 23 observations/leader and 17 observations/team 
member. 

Using the data collected we did an analysis to validate the methodology and the rubrics, namely, what we refer 
to as the instrument. The objective was to determine whether the instrument is a) statistically significant, b) 
consistent with the ability to identify the superior performance, c) discriminates between individuals and 
competencies within the same individual, and d) feasible in our educational model. We have done the statistical 
analysis (a) using the matrix of correlation coefficients (Pearson, 1920; Pearson, 2015) and the Cronbach's alpha 
(Cronbach, 1951), widely used in psychology and social sciences. The consistency (b) of the instrument was 
determined by comparing the results of the instrument with the qualitative appreciation of the students at the 
end of the process. The qualitative appreciation is based on the subjective impression that the observers 
obtained during the year. The observers were asked to separately evaluate only the competencies (not the 
elementary behaviors) on a scale from 0 to 10, considering all the period of the pilot test. Additionally, we 
demanded an external evaluation of the leaders from the supervisor of the internship in industry. We have used 
this evaluation to compare the results with those obtained in the pilot test and the qualitative appreciation. 
These comparisons allow us to say whether these projections of the evaluations are consistent with the results 
obtained from the instrument and the qualitative evaluation. 

Firstly, the participants measured the time spent in completing the corresponding questionnaires (within the 
forms) after the weekly meeting. We found that each team member spent about 10 minutes to evaluate 2 
competencies of her/his remaining 4-5 teammates together with the self-evaluation. The leader spent on the 
order of 20 minutes to evaluate the 5 competencies of her/his 5-6 team members and to complete the self- 
evaluation. In turn, the professors of PLE spent 5 minutes to evaluate one leader for the 5 competencies. 

In Figures 2 a) and b) we show the competency level of two members of the fourth year team (GEQ). In blue, 
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we indicate the results obtained directly from the instrument. In red, we show the qualitative appreciation and, 
in green, the evaluation of the internship (summer 2013, ranging from 2 to 6 months). In this latter case, we 
established a correspondence between the items evaluated in the internship (initiative, communication, 
responsibility, cooperation, etc.) and our social competencies, in order to extract a projection as an estimate of 
the competency level with regards to the dictionary given in Table 1 (Sune, 2015). 

As we observe in both Figures 2 a) and b), the three estimations of the competency level cast a very similar 
profile. Considering the case of Figure la), the qualitative evaluation is numerically equal for the competencies 
Bl, B2, and B5, but shows deviations in B3 and B4. Flowever, taking into account its nature, this evaluation 
tends not to sharply discriminate between the competencies. Rather, it responds to an overall impression of the 
observers about the individual evaluated at the end of the IP. Regarding the internship, the competencies with 
higher scores are B3 and B4, together with B2. Instead, Bl and B5 are lower rated. Hence, we can say that the 
internship evaluation casts a similar profile as the other measures. In Figure 1 b), the evaluations are 
qualitatively coincident. 




Figure 2. Competency level of two members, a) and b), of the team of fourth 
year (GEQ). Comparison between the results of the pilot test , the qualitative 
evaluation and the internship evaluation 


Generally, we observe a good correspondence between the evaluations. However, in some cases, the evaluation 
from the supervisor of the internship overestimates the competencies (not shown), mostly because the person 
filling in the evaluation form is not the person that has directly worked with the student during the internship. 

In Table 3 we indicate the results of the evaluation of one team that presents high variations between the 
evaluations of its team members. The code of colors used in Table 3 corresponds to: red, for a competency level 
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clearly to be improved (lower than 6); yellow, that the competency level is average (between 6 and 8.25), and, 
green, that the competency level is excellent (higher than 8.25). These are the qualitative levels used in the 
assessment. As we can observe in Table 3, TM2 has the worst results of the team and TM4 has the best results, 
particularly in the B3 competency. The tutor of the team suggested, according to his/her experience, that TM4 
was a hard worker and he/she interacted a lot with the teammates. Actually, he/she was acting as a leader 
among the team members in the absence of the formal leader, or during the ongoing tasks. Furthermore, the 
tutor also indicated that TM2 had no commitment with the team and he/she did not deliver, for example, the 
outcomes of the tasks on time. 



Team members 

Leader 

Competency 

TM1 

TM2 

TM3 

TM4 

TM5 

L 

81. Human Interaction <& Versatility 

6.65 

5.95 

6.89 

8.63 

6.11 

7.92 

82. Facilitative Leadership 

5.89 

5.27 

6.61 

7.56 

6.36 

8.48 

83. Teamwork 

5.66 

5.45 

5.92 

8.30 

6.30 

7.18 

84. Active Learning & Responsibility 

6.22 

5.57 

6.42 

7.78 

7.18 

7.78 

B5. Initiative & Innovation 

6.02 

3.78 

6.91 

6.12 

5.44 

7.32 


Table 3. Results of the competency evaluations of a given team. Red indicates that the 
competency level is lower than 6; yellow indicates that the competency level is between 6 and 
8.25, and green indicates that the competency level is higher than 8.25 


The interview with the tutor revealed that this team had some difficulties to carry out the project. The 
evaluations of other teams show different results whose detailed analysis, however, go along the same lines as 
the case exposed here. Therefore, from this analysis we can conclude that the instrument is able to respond to 
the set objectives. 

From a statistical point of view, the matrix of correlation coefficients (Pearson, 2015) is defined as 


r Ax°A/ 

"'■lUFfilF? 111 

where x a and are two observation of given pinpointed behaviors related to competency X and Y, respectively, 
of the same individual. The bars indicate an average over the population and Ax a stands for the deviation with 
respect to the mean. In Table 4 we present the correlations between the elementary behaviors related with the 
B4 competency, which is part of the 37 x 37 team members' correlation matrix. The negative correlations 
appear in magenta. If the element is higher than 0.5, it appears in red. The negative correlations indicate grosso 
modo, that, the two correlated behaviors on average occur in opposition. If the coefficient is higher than 0.5 
(and positive), it means that both correlated behaviors are simultaneously over the mean. High correlation 
coefficient may indicate redundancies in the questionnaire. As an example, the coefficient 0.577 that appears in 
Table 4, is referred to the following pair rubrics: 

i26: Explains what new knowledge has been acquired from books, articles, web sites, conversations with 
professors, etc. related to the course subjects or to the development of the project. 

i27: The delivered reports, posters, etc. contain well quoted references that have been used to produce 
the outcome. She/he can qualitatively explain what the content of the cited reference is. 

Although the coefficient is high, we can see from their phrasing that these two elementary behaviors are not 
redundant. The correlation is hence associated to a trait of the population, namely, that a person who (i26) 
Explains what new knowledge... typically also (i27) The delivered reports, posters... 
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i23 

i24 

i25 

i26 

i27 

i28 

i29 

i30 

i23 

1.000 

0.045 

0.114 

0.280 

0.304 

-0.077 

0.417 

0.249 

i24 

0.045 

1.000 

0.458 

0.034 

0.167 

0.361 

0.007 

0.391 

i25 

0.114 

0.458 

1.000 

-0.023 

0.141 

0.382 

0.013 

0.340 

i26 

0.280 

0.034 

-0.023 

1.000 

0.577 

0.130 

0.398 

0.015 

i27 

0.304 

0.167 

0.141 

0.577 

1.000 

0.137 

0.310 

0.140 

\28 

-0.077 

0.361 

0.382 

0.130 

0.137 

1.000 

0.144 

0.412 

i29 

0.417 

0.007 

0.013 

0.398 

0.310 

0.144 

1.000 

0.121 

i30 

0.249 

0.391 

0.340 

0.015 

0.140 

0.412 

0.121 

1.000 


Table 4. Matrix of correlation coefficients (Pearson, 2015) of the B4 competency. The coefficients 
are within the range [-1,1]. The rubrics related to B4 competency are from \23 to i30 


Finally, Cronbach's alpha can be calculated from the variances. The maximum value of Cronbach's alpha is 1 and 
indicates that the internal consistency of the questionnaires is excellent. The values of Cronbach's alpha 
obtained for the questionnaire of the leader and team member are 0.81 and 0.87, respectively. This fact allows 
us to conclude that the instrument consistently measures the intended competency level. 

4 CONCLUSIONS 

From the analysis of the results we see that the instrument is statistically consistent and permits one to identify 
competency levels appearing in professional situations. At the same time, it allows one to discriminate not only 
between members of the same team, but also between competencies of the same individual. The predictive 
capability of the instrument for superior performance, in the sense of our quotation of Spencer and Spencer 
(Spencer & Spencer, 1993), has been checked only from the internship of the fourth year students. The 
consolidation of the instrument within the educational model of the ETSEQ should allow carrying out long-term 
studies about the professional evolution in the labor market over 5 or 10 years. 

The positive results of the pilot test allowed the implementation of the methodology of evaluation of social 
competencies to all years in the Bachelors of Chemical Engineering and Food Engineering, during the academic 
year 2013-2014. This implementation demanded the participation of more than 300 students, 7 coordinators 
and more than 15 tutors. The treatment of the data collected, the maintenance of confidentiality, and the 
necessity of coordinating a large number of individuals are the main obstacles to effectively implement the 
instrument and, more importantly, make the methodology sustainable. These problems can be solved using 
specific tailor-made software to enable remote access, keep the data secure, treat the information 
automatically, and manage confidentiality. On the other hand, our experience suggests that a reduction of the 
items to be evaluated is advisable, and that the range of values per item should be increased. Both actions aim 
at simplifying and systemizing the procedure in order to guarantee the sustainability of the methodology. 

Secondly, the implementation of this methodology of assessing social competencies reveals new information 
about our students. This knowledge is useful for the students since they are confronted with what the others 
see in them, in the context of the IP: yvooBi oeauiov, know thyself. Such knowledge should trigger by itself 
behavioral changes in the students. In the first place, the exposure to the rubrics, when acting as evaluators, 
causes behavioral changes by imitation of actions suggested by the standard, as we observed during the pilot 
test. In the second place, the assessment places the students face-to-face with what others see in them. If they 
dislike the image returned by this mirror they will have motivation to change. Furthermore, the existence of this 
objective knowledge about the students introduces a new educational necessity in the staff: how to help our 
students to improve their competency level before they are ready to enter the labor marked. Therefore, 
including improvement plans in our educational model will be a subject of major research in the future. 
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